Overview

This report provides an evaluation of Experimental Target forecasts of confirmed Influenza Hospitalization Admissions for the 2022-2023 season to the CDC FluSight Forecast Hub. Experimental target submissions began on December 12, 2022. The purpose of introducing an additional experimental target is to provide an opportunity for forecasting teams to submit forecasts for increasing and decreasing activity. The experimental target, named “2 wk flu hosp rate change” will be submitted as estimates of the probability of occurrence for each rate change category. Guidance for submission of the experimental target is shown here.

This report evaluates experimental forecasts at the state and national level for confirmed Influenza Hospitalization Admissions for the 2022-2023 season. Data from the HealthData is used as ground truth data for evaluating the forecasts.

We evaluate models based on the Brier Score. To account for the variation in difficult of forecasting different weeks and locations, a pairwise approach was used to calculated the relative adjusted Brier score. Models with relative scores lower than 1 have been more accurate than the baseline on average, whereas relative scores greater than 1 indicate less accuracy than baseline on average.

Three models are used as baseline for the pairwise comparison. Two models from the github repository for CDC FluSight hospitalization are used Flusight-baseline and Flusight-ensemble. We used a version of this code to convert these into experimental target forecasts.

Results

The table evaluates forecast models based on the relative Brier Score, aggregated across weeks and location.

Inclusion criteria for each column are detailed below the table.

Summary Tables

The models included started submitting forecasts on December 12, 2022. The models included have submitted at least 50% of forecasts during this time, where one forecast is a location, target, forecast date combination. The data are initially ordered by model based on their average relative Brier score aggregated across time and location, with the most accurate models at the top.

  • The column titled, “# forecasts” lists the number of forecasts a team has submitted.
  • Columns 3 through 5 calculate the adjusted relative Brier score based on the three baseline models used for pairwise comparison.
  • Column 6 calculates the average relative Brier score across the pairwise comparisons of the three baseline models.

Evaluation by Week

In the following figures, we have evaluated models across multiple forecasting weeks. Points included in this comparison are for all models that have submitted experimental target forecasts. The models in the legend with a dot and line have scores for every week. The models with just a line are missing scores for at least one week.

Brier score is used as a metric. The figure shows the mean Brier score across all locations for submission weeks.

Evaluation Periods

This figure shows the number of confirmed Influenza Hospitalization Admissions reported each week in the US for the 2022-2023 season. .